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BOTTLENECK  PROBLEMS  AND  DYNAMIC  PROGRAMMING 
Rlchax^  BelltMn 


§1.  Introduction. 

^Vrhe  purpose  of  tMs  paper  Is  to  Indicate  how  the  theory  of 
dynamic  programnlng  provides  a  mathematical  formulation  and  a  sys¬ 
tematic  approach  to  an  Interesting  and  significant  class  of  pro¬ 
duction  and  allocation  problems,  which  we  ahall  call  "bottleneclr*^ 

'two  problems 


Problem  1; 

At  some  Initial  period  we  possess  a  quantity  Xt  of  steel  and 
a  capacity  Xc  of  steel  production,  with  the  privilege  of 
dividing  the  qpiantlty  Xt  Into  three  parts,  yt ,  yt,  ys  where  yi 
Is  to  be  used  to  Increase  the  capacity,  ya  to  be  used  together 
with  the  present  capacity  to  produce  more  steel,  and  ya  to 
remain  In  the  stockpile.  Given  that  this  allocation  and  pro¬ 
duction  process  continues  for  a  fixed  number  of  time  periods 
and  given  the  Increase  In  capacity  determined  by  Xt  and  yi  and 
the  Increase  In  the  quantity  of  steel  determined  by  Xa  and  ya, 
the  question  Is  to  determine  the  allocation  policy  which  maxi¬ 
mises  the  quantity  of  steel  In  the  stockpile  at  the  end  of 
the  final  period. 

Problem  2: 

We  are  engaged  in  the  manufacture  of  an  Item  which  requires 
two  eosq)onent  parts.  We  possess  resources  which  may  be  used 
to  increase  the  rates  of  production  of  these  Items.  Assuming 
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that  we  know  the  relation  between  rates  of  production  and 
the  rate  of  allocation  of  resources,  what  policy  of  allo¬ 
cation  of  resources  do  we  employ  in  order  to  aaziaise  the 
output  of  conplete  iteas  over  a  fixed  tine  interval? 

These  problems  in  their  discrete  forms  where  decisions  are 
made  at  a  finite  number  of  fixed  times  nay  in  sosie  cases  be  attacked 
by  the  cosqnjtational  techniques  of  the  theories  of  linear  and  non¬ 
linear  prograsming.  However,  not  only  do  the  simplest  problems, 
taken  over  time  intervals  of  quite  reasonable  length,  give  rise  to 
matrices  of  unreasonable  dimensions  but,  more  serious  from  the 
esthetic  and  mathematical  point  of  view,  this  approach  takes  no 
cognisance  of  the  intrinsic  structure  of  the  process. 

If  we  consider  the  problem  in  its  continuous  form  (an  approxi¬ 
mation  technique  which  almost  always  results  in  a  great  analytic 
simplification),  simple  assumptions  of  linearity  yield  the  system 
of  differential  equations 


"  *171*  *i(0)  ■  C|,  (1) 

-  -Xi-ya  ♦  Min  (b,X|,a«ya)  -  (««-l)ys-y»»  x«(0)  •  c«, 

with  yj  and  yc  subject  to  the  restrictions 


(•)  yi  »ys  ^  0 
(b)  yf»-y»  i  *« 

(=)  Tt  < 


U) 
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T)»e  tern  Mln(btXt  tacye )  represents  the  bottleneck  condition  snd 
Is  the  nsthemstloml  equivalent  of  the  1 Imitation  on  the  growth  of 
the  steel  stockpile  due  to  Insufficient  capacity.  Our  aim  Is  now 
to  nazinlse  xc(T). 

In  this  fonn  the  problea  nay  be  attacked  by  classical  nethods 
of  the  calculus  of  variations  utilizing  Lagrange  multipliers.  How- 
ever»  once  again  this  nethod  is  too  general  and  the  particular 
features  of  the  problem  are  overshadowed. 

A  third  attack  on  problems  of  this  genre  Is  afforded  by  the 
very  Interesting  techniques  developed  by  R.  Isaacs  In  his  study  of 
a  class  of  continuous  games.  Although  there  are  some  points  of 
contact  between  our  methods,  the  conceptual  bases  and  the  analytic 
continuations  are  quite  distinct.  In  particular,  his  techniques 
would  seem  to  lead  to  grave  computational  difficulties. 

Let  us  mention  that  a  simple  Interpretation  of  the  second 

problem  above  leads  to  the  mathematical  question  of  maximizing 
T 

J*  Mln(x,y)dt  over  all  ft  and  fs  satisfying  0  ^  fi,f8  ^  M, 
o 


(ft'«'fs)dt  ^  Ca,  where 


•  hiX  ♦  a*y  +  f»  ,  x(0)  ■  Cl 

-  asx  -»•  a4y  ft,  (0)  -  c*. 


(3) 


The  Dynamic  Progratlng  Approach. 

We  ahall  consider  only  the  first  problem  In  discussing  the 
application  of  dynamic  programming  techniques.  For  a  brief  dls— 
cusalon  of  the  theory  and  some  typical  problems  we  refer  to  [i],  [a]. 
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Our  first  step  Is  to  Irobsd  the  problem  within  s  continuous 
family  by  taking  the  quantities  Xi ,  Xs  and  T,  the  duration  of  the 
process,  as  the  basic  state  variables.  The  effect  of  any  deei— 
Sion,  where  by  a  decision  we  mean  a  choice  of  yt ,  vz  and  ys,  will 
be  to  transform  these  state  variables  into  a  similar  triple.  The 
criterion  function  Max  Xc(T)  smy  be  written  f(xi,X2,T}.  Using 
the  intuitive  and  obvious  property  that  an  optimal  policy  has  the 
property  that  its  continuation  after  any  initial  decisions  must 
also  be  optimal  with  respect  to  the  new  state  variables,  we  obtain 
the  characteristic  functional  equation  of  dynamic  programming 


f  (ci  ,ct  ,e-f  t ) 

f (c  1  ,C2 ,0 ) 


Max  f(xi(s),xa(a),t), 

Cz  »  Cl  ,Ca  ^  0, 


(1) 


where  Xt  and  Xa  satisfy  (l)  of  ol  and  by  Max  we  mean  that  the 

maximum  is  to  be  sought  over  all  yi  and  ya  which  satisfy  the  res¬ 
trictions  in  (2)  of  ^  over  .  It  is  quite  easy  to  prove  by 

elementary  arguments  that  the  maximum  is  assumed.  Furthermore,  it 
follows  readily  from  (l)  that  the  solution  is  unique.  For  setting 
t  •  0,  we  obtain,  using  the  initial  condition 

-  Ita  f(»t(»),»«(»),0)  -  Mm,  ».(•)  (a) 


Hence  the  functional  equation  is  equivalent  to  the  original  pro¬ 
cess.  Alternatively,  it  is  easy  to  establish  directly  by  succeaeive 
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•pproxlBfttlons  that  (l)  has  a  solution. 

Asaualng  that  f  has  «ell-t>ehave<3  partial  derivatives  with 
respect  to  the  basic  variables,  we  obtain  via  a  Halting  process, 
as  s  ■»  0,  the  partial  differential  equation 

[*■='*  -Ur  ]  (3) 

where  by  D^O]  we  mean  the  region  In  yi  ,ys  space  described  by 

Oiyt.ya.  yi+ya  i  C»,  ya  1  biCt/aa  (^) 

Since  It  seems  rather  difficult  to  prove  directly  that  the  solu— 
tlon  of  (1)  possesses  the  requisite  continuity  properties,  an 
Indirect  approach  Is  used.  It  Is  first  shown  without  difficulty 
that  any  solution  of  (3)  satisfies  (2).  Having  obtained  a  solu¬ 
tion  of  (1)  by  the  use  of  (3),  the  uniqueness  theorem  tells  us 
that  It  Is  the  solution  of  (l). 

Turning  to  the  task  of  finding  solutions  of  (3)  we  first  con¬ 
sider  the  case  of  small  t  where  the  solution  Is  lamed late.  Having 
obtained  f  for  small  t,  the  partial  differential  equation  Is  now 
utilized  to  deduce  the  form  of  solution  as  t  grows.  Since  this  Is 
a  "bootstrap"  method  It  Is  necessary  to  verify  with  great  care 
that  the  f  obtained  in  this  fashion  Is  actually  a  solution  of  (3). 
Having  found  f  we  can  now  determine  whether  or  not  the  optimal 
policy  Is  unique.  It  turns  out  to  be  so  In  this  case. 
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^4.  An  Outline  of  the  Analytic  Procedure. 

Let  U8  give  a  short  sketch  of  the  steps  to  be  followed  in 
obtaining  the  solution  of  the  maximization  problem.  WS  shall  con¬ 
sider  the  most  complicated  case  where  no  capacity  restriction  exists 
Initially,  l.e.,  ca  <  bjCi/aa.  For  processes  of  short  duration  the 
solution  is  clear,  yi  *0,  y*  •  Xa,  yielding  f  -  cae^**“'^^. 

This  policy  Is  pux*8ued  until  a  bottleneck  develops.  Using  the 
above  allocation  program,  this  will  occur  as  soon  as  t  exceeds 
T»  -  (log  b»Ci/a«ca)/(aa-l). 

Ih  order  to  obtain  tlw  solution  for  tines  greater  than  Tt# 

a 

rewrite  the  right  side  of  (3)  of  §3  In  the  fons 

*  S!S]  L^‘‘“  ^  ^  ^ 

The  location  of  the  maximizing  point  (yi(0),  ya(0))  will  depend 
upon  the  sign  and  magnitude  of  the  coefficients  of  yi  and  ya.  For 
t  <  Ti,  the  coefficient  of  yi  Is  uniformly  negative,  while  that 
of  ya  Is  unlfonnly  positive. 

Proceeding  on  the  assumption  of  continuity,  we  suspect  that 
the  solution  for  t  slightly  greater  than  T|  will  have  the  fora 
71-0,  ya  -  xa  for  0  ^  s  ^  Tt  and  yi  -  0,  ya  -  btXi/sa  (due  to 
the  capacity  restriction)  for  T|  ^  s  ^  t. 

The  function  f  will  now  have  the  form 
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In  order  to  find  out  how  long  this  policy  endures,  we  apply  (3) 
of  §3  at  the  point  s  ■  Tj.  Starting  from  this  point  me  see  that 
f  has  tha  for«  ««  4-  1)  htOi  ( t— Ti  )/a«. 

lha  eoafflelant  of  Jt  in  (l)  la  at  >f/^ot  -  ^/doa 
•  aib| (a»— 1 ) ( t— Ti )/a8  -  1.  This  Is  0  at  t— Ti  •  aa/aj bj (ag-l )  •  T*. 

This  shows  that  this  modified  policy  Is  optimal  for  Ti  t  ^  Ti 
+  aa/sibi  (aa— 1 ) . 

For  larger  t,  there  will  be  an  Interior  Interval  during  wnlch 
yt  -  Xa— biXi/ae.  This  corresponds  to  allocating  steel  to  Increase 
the  capacity  of  steel  mills.  For  large  t,  the  optimal  policy  has 
the  form 

yi  -  0,  ya  "  Xa,  0  ^  s  ^  Ti 
yi  -  xz 
yi  -  0, 

where  T*  Is  as  above. 

Since  we  have  pursued  a  bootstrap  method  it  is  essential  as 
mentioned  above  that  this  solution  be  verified.  With  application 
to  Inter— Industx^  problems  In  mind,  it  Is  vital  that  this  verifi¬ 
cation  be  analytic  rather  than  numerical.  The  details  of  the  veri¬ 
fication  are  not  trivial  and  we  shall  postpone  any  discussion  until 
a  future  time. 

§5-  Discussion. 

The  method  we  have  presented  in  skeleton  form  Is  applicable 
to  a  large  class  of  problems  related  to  the  maximization  of  functions 


yz 


b|Xi 

As 


Ti  <  a  ^  t  -  T 


(3) 


yz  - 


t  -  T*  ^  8  i  t. 
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or  functionals  depending  upon  the  solutions  of  differential 
equations.  It  Is  also  applicable  to  various  classes  of  multi¬ 
stage  continuous  games,  such  as  games  of  survival. 

If — as  Is  necessary  In  more  realistic  mathematical  models 
dealing  with  the  production  of  capital  goods — time  lags  are  taken 
Into  account,  the  complexity  of  the  problem  Increases. 

A  complete  and  detailed  treatment  of  the  above  problem  will 
be  presented  subsequently,  together  with  a  discussion  of  extensions 
In  the  directions  Just  clt‘»d. 
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